What does this PR do?

• Adds a ECS Logs integration aiming at simplifying the life of users who want to ingest a log file that's produced by one of our ECS logging libraries.
• Currently uses the logfile input (should use filestream later)
  • Uses defaults optimized for ECS JSON log files:
  • Enables all relevant metadata processors
  • Enables multiline settings appropriate for the stackTraceAsArray setting of ecs-logging-java
  • Excludes .gz files by default from file patterns that include a wildcard at the end to match rolled over files. Example: app.log, app.log.1, app.log.2.gz
• Logs are sent to the logs-ecs_router-default data stream
  • This data stream has a default ingest pipeline which does the following things:
    • Parsing the JSON
    • Routing the logs to a different data stream, based on the content of the log: logs-${data_stream.dataset:generic}-${data_stream.namespace:default}. So if the logs contain data_stream.dataset: foo (or event.dataset: foo), they will be sent to logs-foo-default.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

TODOs/open questions

• Use filestream instead of logfile input
• Add default multi line setting for the stackTraceAsArray option (if line does not start with {)
• Use include/exclude to include rolled over files but exclude .gz
• Should we build on top of [Filestream] New Input package #2735 so that we only apply defaults (dataset, processors, multiline, include/exclude)?
• End-to-end test with local agent

Blocked by

• [Change Proposal] Ability to add privileges for all data streams of a specific type package-spec#315

Follow ups

• Handle mapping conflicts
  • Custom fields conflict with ECS fields. Corrupts the mapping so that docs with proper ECS fields fail. Hard to recover from. Optionally use ECS conventions for dynamic mappings elasticsearch#85692
  • Custom fields conflicting with other custom fields, such as foo foo.bar (foo used both as scalar and object type). https://github.com/elastic/observability-dev/issues/1661
• Data stream router processor to avoid expensive mustache script Add reroute processor elasticsearch#76511

Author's Checklist

How to test this PR locally

Related issues

• closes [Feature Request] Enable ECS log detection by default using Custom Log integration #1332

Screenshots